<html>
<head><meta charset="utf-8"><title>Gathering parallel data · t-compiler/wg-parallel-rustc · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/index.html">t-compiler/wg-parallel-rustc</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html">Gathering parallel data</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="165560684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165560684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165560684">(May 13 2019 at 19:24)</a>:</h4>
<p>So we talked about trying to gather various bits of parallel data -- do we think we'll be able to do that by this Friday? It seems ok if not, but we should just put off the meeting</p>



<a name="165583579"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165583579" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165583579">(May 14 2019 at 01:02)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> I'm waiting on flags from <span class="user-mention" data-user-id="116466">@Zoxc</span>; I hope to start the implementation groundwork tomorrow morning. I am optimistic for Friday having relevant stats, though maybe not all (depending in part on how quickly I can get it done and get eddyb and you to run relevant commands)</p>



<a name="165583749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165583749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165583749">(May 14 2019 at 01:06)</a>:</h4>
<p>You can use <code>-Z threads=n</code> for rustc and <code> -j n</code> for cargo</p>



<a name="165584237"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165584237" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165584237">(May 14 2019 at 01:16)</a>:</h4>
<p><span class="user-mention" data-user-id="116466">@Zoxc</span> Should I use both simultaneously? Or test with -j{1,2,4,8} and -Zthreads={1,2,4,8} with both varying independently?</p>



<a name="165584248"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165584248" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165584248">(May 14 2019 at 01:16)</a>:</h4>
<p>er, simultaneously meaning "same n"</p>



<a name="165587253"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165587253" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165587253">(May 14 2019 at 02:33)</a>:</h4>
<p>Good question. I think just setting <code>-Z thread</code> will suffice. Using <code>-j1</code> might be more accurate if measuring overhead though</p>



<a name="165588612"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165588612" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165588612">(May 14 2019 at 03:03)</a>:</h4>
<p>okay, will investigate</p>



<a name="165588617"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165588617" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165588617">(May 14 2019 at 03:04)</a>:</h4>
<p>Depending on time will maybe try to do both</p>



<a name="165624730"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165624730" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165624730">(May 14 2019 at 14:00)</a>:</h4>
<p>Are we testing parallel-rustc on any arbitrary crates/rust projects? Or we can narrow the list to some representative ones? This question comes from a confusion of mine that when we say we need to do experiment on parallel-rusts, is that means we need to guarantee the parallel-rustc will not add significant overhead to compiling any crates? Or actually we primarily want to have the parallel-rustc to perform consistently on any kinds of hardware (CPU architectures, num of cores/threads, etc.). I believe we should do both. And those two are actually orthogonal with each other, so that we can do them separately.</p>



<a name="165625424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165625424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165625424">(May 14 2019 at 14:10)</a>:</h4>
<p>We are testing it for now on perf.r-l.o and hopefully some other hardware (but that is a bit up in the air); with the set of crates we use there and the hardware we use there</p>



<a name="165625539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165625539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165625539">(May 14 2019 at 14:12)</a>:</h4>
<p>That make sense. I want to know the plan further. When we move on to the public experiment phases. I believe we want to have comparison data for both kinds of testing I just mentioned.</p>



<a name="165625863"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165625863" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165625863">(May 14 2019 at 14:16)</a>:</h4>
<p>Oh, yeah, definitely -- I don't think we've established exactly what our goals are but we would want to test on different hardware/OS/etc and such</p>



<a name="165625999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165625999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165625999">(May 14 2019 at 14:18)</a>:</h4>
<p>Have we got enough confidence from perf.rlo so that we can move on to the following phases?</p>



<a name="165626864"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165626864" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165626864">(May 14 2019 at 14:28)</a>:</h4>
<p>It's our best bet at this point for initial evaluation and should be relatively representative IMO</p>



<a name="165626891"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165626891" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165626891">(May 14 2019 at 14:28)</a>:</h4>
<p>at least for these purposes</p>



<a name="165649371"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165649371" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165649371">(May 14 2019 at 18:40)</a>:</h4>
<p><span class="user-mention" data-user-id="116466">@Zoxc</span> What rustc/cargo have the parallel-rustc feature? I've tried nightly and local build of master branch, neither has the feature.</p>



<a name="165649589"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165649589" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165649589">(May 14 2019 at 18:42)</a>:</h4>
<p>You need to set [rust] parallel-compiler = true in config.toml</p>



<a name="165650946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165650946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165650946">(May 14 2019 at 18:58)</a>:</h4>
<p>should I clean the previous build? I just tried with adding such config. And use <code>./x.py build -i --stage 1</code>, the feature is still not available. I may get it work with a completely new compilation of rustc. But it takes long time (about 1 hour on my i5 desktop). I just want to avoid some redundant compilation.</p>



<a name="165651632"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165651632" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165651632">(May 14 2019 at 19:04)</a>:</h4>
<p>Yeah, you need to do x.py clean</p>



<a name="165653700"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165653700" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165653700">(May 14 2019 at 19:26)</a>:</h4>
<p>While I'm waiting for the compilation, I just fill some info in the HackMD (<a href="https://hackmd.io/KmHulVmISKu7L2HmNgbPgg?edit" target="_blank" title="https://hackmd.io/KmHulVmISKu7L2HmNgbPgg?edit">https://hackmd.io/KmHulVmISKu7L2HmNgbPgg?edit</a>) which was created by <span class="user-mention" data-user-id="116009">@nikomatsakis</span> last week. Niko mentions "Mark" to provide some guide on configuration/setup information. Which Mark should we call? I'm new here and don't want to bother some one else.</p>



<a name="165654421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165654421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165654421">(May 14 2019 at 19:36)</a>:</h4>
<p>That would be <span class="user-mention" data-user-id="116122">@simulacrum</span></p>



<a name="165654609"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165654609" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165654609">(May 14 2019 at 19:38)</a>:</h4>
<p>I've verified that the parallel compiler works (i.e., can build all of perf) and have initial stats here <a href="https://perf.rust-lang.org/compare.html?start=80e7cde2238e837a9d6a240af9a3253f469bb2cf&amp;end=6f087ac1c17723a84fd45f445c9887dbff61f8c0" target="_blank" title="https://perf.rust-lang.org/compare.html?start=80e7cde2238e837a9d6a240af9a3253f469bb2cf&amp;end=6f087ac1c17723a84fd45f445c9887dbff61f8c0">https://perf.rust-lang.org/compare.html?start=80e7cde2238e837a9d6a240af9a3253f469bb2cf&amp;end=6f087ac1c17723a84fd45f445c9887dbff61f8c0</a> but they're mostly meaningless (i.e., no -Zthreads and such was specified -- so I guess this is the -j4 with -Zthreads=1 but not confident that's correct). I will be collecting further stats over tonight and tomorrow hopefully</p>



<a name="165654909"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165654909" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165654909">(May 14 2019 at 19:42)</a>:</h4>
<p>That should be -j8 + Zthreads=8, <a href="https://perf.rust-lang.org/compare.html?start=80e7cde2238e837a9d6a240af9a3253f469bb2cf&amp;end=6f087ac1c17723a84fd45f445c9887dbff61f8c0&amp;stat=wall-time" target="_blank" title="https://perf.rust-lang.org/compare.html?start=80e7cde2238e837a9d6a240af9a3253f469bb2cf&amp;end=6f087ac1c17723a84fd45f445c9887dbff61f8c0&amp;stat=wall-time">https://perf.rust-lang.org/compare.html?start=80e7cde2238e837a9d6a240af9a3253f469bb2cf&amp;end=6f087ac1c17723a84fd45f445c9887dbff61f8c0&amp;stat=wall-time</a></p>



<a name="165655387"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165655387" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165655387">(May 14 2019 at 19:47)</a>:</h4>
<p>ah okay so the default is num_cpus for threads, got it</p>



<a name="165745198"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165745198" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165745198">(May 15 2019 at 19:15)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> so is there something you want me or <span class="user-mention" data-user-id="119009">@eddyb</span> to test?</p>



<a name="165745210"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165745210" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165745210">(May 15 2019 at 19:15)</a>:</h4>
<p>and do we think we'll have the set of data we were looking for ready? (or some subset?)</p>



<a name="165750778"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165750778" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165750778">(May 15 2019 at 20:21)</a>:</h4>
<p>We should have at the least a very good set of data</p>



<a name="165750826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165750826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165750826">(May 15 2019 at 20:21)</a>:</h4>
<p>I want to confirm that the data I've currently gathered is correct (i.e., I'm gathering good data and not just spinning CPU cycles) and hope to get you two a script by tonight that'll dump relevant data in a directory</p>



<a name="165750891"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165750891" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165750891">(May 15 2019 at 20:22)</a>:</h4>
<p>(But even if we don't get data from <span class="user-mention" data-user-id="119009">@eddyb</span> or you I think we will have enough to discuss)</p>



<a name="165764969"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165764969" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165764969">(May 15 2019 at 23:37)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> <span class="user-mention" data-user-id="119009">@eddyb</span> If I could get you to run <a href="https://gist.github.com/Mark-Simulacrum/b5dd679b03ad3b979ffad2dfea2a7efc" target="_blank" title="https://gist.github.com/Mark-Simulacrum/b5dd679b03ad3b979ffad2dfea2a7efc">https://gist.github.com/Mark-Simulacrum/b5dd679b03ad3b979ffad2dfea2a7efc</a> that'd be great; I would edit <a href="https://gist.github.com/Mark-Simulacrum/b5dd679b03ad3b979ffad2dfea2a7efc#file-collect-parallel-data-sh-L39" target="_blank" title="https://gist.github.com/Mark-Simulacrum/b5dd679b03ad3b979ffad2dfea2a7efc#file-collect-parallel-data-sh-L39">this line</a> to go up to the number of cores you have</p>



<a name="165765037"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165765037" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165765037">(May 15 2019 at 23:38)</a>:</h4>
<p>this will take approximately 2ish hours per "run" at least on perf.r-l.o's collector; the script as-is takes around 10 hours or so</p>



<a name="165765054"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165765054" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165765054">(May 15 2019 at 23:39)</a>:</h4>
<p>if we don't get data though I've updated the doc with the data I've already collected</p>



<a name="165765055"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165765055" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165765055">(May 15 2019 at 23:39)</a>:</h4>
<p><a href="https://hackmd.io/KmHulVmISKu7L2HmNgbPgg?both#Measurements" target="_blank" title="https://hackmd.io/KmHulVmISKu7L2HmNgbPgg?both#Measurements">https://hackmd.io/KmHulVmISKu7L2HmNgbPgg?both#Measurements</a></p>



<a name="165765056"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165765056" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165765056">(May 15 2019 at 23:39)</a>:</h4>
<p>That is not whole-crate-graph since I haven't had time to do that but is at least a start</p>



<a name="165795624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165795624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165795624">(May 16 2019 at 10:03)</a>:</h4>
<p>thanks for gathering that data, <span class="user-mention" data-user-id="116122">@simulacrum</span>!</p>



<a name="165795704"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165795704" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165795704">(May 16 2019 at 10:04)</a>:</h4>
<p>I think we need whole-crate-graph data too, since that is the case where the jobserver comes into play</p>



<a name="165802709"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165802709" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165802709">(May 16 2019 at 11:59)</a>:</h4>
<p>Yeah, I'm working on some manual runs</p>



<a name="165802932"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165802932" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165802932">(May 16 2019 at 12:02)</a>:</h4>
<p>though I suspect the reality might be that whole-crate graph is quite rare since most people are incrementally recompiling 1-2 crates at the end</p>



<a name="165804264"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165804264" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165804264">(May 16 2019 at 12:21)</a>:</h4>
<p>for local builds yes, but for CI builds it's different</p>



<a name="165804516"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165804516" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165804516">(May 16 2019 at 12:24)</a>:</h4>
<p>true -- I could imagine us having Cargo try to detect how many crates it's building and set different -Z flags based on that</p>



<a name="165804525"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165804525" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165804525">(May 16 2019 at 12:24)</a>:</h4>
<p>(well, -Cthreads= I guess)</p>



<a name="165820501"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165820501" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165820501">(May 16 2019 at 15:26)</a>:</h4>
<p>Just read those compelling metrics. I'm thinking about a confusion about the threshold. Do we really care the overhead when run with "-Zthreads=1"?  This question is somehow important when it comes to how will the compiler behave in the future. </p>
<p>In my envision, the compiler may automatically pick a sensible number of threads for parallel query. As long as the number is greater than 1,  parallel-rustc will provide significant performance gain to users. </p>
<p>When rustc is running on an one-core/one-thread machine, can we make the compiler run as current "single" thread version instead of the real "-Zthread=1" which introduce overhead? Here, I'm only talking about the technical capability. Of course if the user explicitly set "-Zthreads=1", we should let the compiler run as the real parallel compiler even though only one thread can be used.</p>
<p>If that is the case, then I believe we can definitely move on to the public experimental phases. Users can only encounter bad overhead when they force the compiler to run with "-Zthreads=1" which we will notify the users in the announcement blog.</p>



<a name="165833141"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833141" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833141">(May 16 2019 at 18:00)</a>:</h4>
<p><span class="user-mention" data-user-id="220056">@lwshang</span> we do care about the overhead with -Zthreads=1, if for no other reason than sometimes there is only one core avail</p>



<a name="165833215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833215">(May 16 2019 at 18:00)</a>:</h4>
<p>In order to eliminate that overhead, we'd effectively have to ship two copies of the compiler</p>



<a name="165833219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833219">(May 16 2019 at 18:00)</a>:</h4>
<p>One compiled to avoid locks etc</p>



<a name="165833221"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833221" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833221">(May 16 2019 at 18:00)</a>:</h4>
<p>and one not</p>



<a name="165833225"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833225" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833225">(May 16 2019 at 18:00)</a>:</h4>
<p>not likely feasible</p>



<a name="165833239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833239">(May 16 2019 at 18:00)</a>:</h4>
<p>besides, the more we can optimize that overhead, the better for everyone</p>



<a name="165833282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833282">(May 16 2019 at 18:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> i'm giving your executable a try :)</p>



<a name="165833285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165833285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165833285">(May 16 2019 at 18:01)</a>:</h4>
<p>your script, that is</p>



<a name="165838749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165838749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165838749">(May 16 2019 at 18:59)</a>:</h4>
<p>As shown in the HackMD file, "-Zthreads=1" introduce a overhead about 10%~20%. Our consensus on the threshold suggest that such overhead should be smaller than 5%. Therefore, may I say that the parallel-rustc feature is not ready considering performance with "-Zthreads=1". </p>
<p>Further, I have several questions following. Should we now focus on improving the implementation? Do we know how to improve it? Can we complete such improvement under our current roadmap?</p>



<a name="165857628"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165857628" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165857628">(May 16 2019 at 23:17)</a>:</h4>
<p><span class="user-mention" data-user-id="116466">@Zoxc</span> mentioned some global locks that I suspect we can eliminate or at least look at measuring and reducing contention</p>



<a name="165857707"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165857707" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165857707">(May 16 2019 at 23:18)</a>:</h4>
<p>I plan to have rustc stage 0 timing data with/without parallel compiler as bootstrap and with various thread configurations; I am also currently doing  for Cargo. Depending on when that completes, I will try to get servo, cranelift, and/or ripgrep started so that it's ready by tomorrow morning</p>



<a name="165862574"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165862574" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165862574">(May 17 2019 at 00:49)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> presuming that this process worked (don't see any <em>obvious</em> errors) the data files are at <a href="http://smallcultfollowing.com/perf-data.tar.gz" target="_blank" title="http://smallcultfollowing.com/perf-data.tar.gz">http://smallcultfollowing.com/perf-data.tar.gz</a></p>



<a name="165869080"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165869080" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165869080">(May 17 2019 at 03:22)</a>:</h4>
<p>Will get those uploaded tonight or tomorrow morning. Thanks!</p>



<a name="165869090"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165869090" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165869090">(May 17 2019 at 03:23)</a>:</h4>
<p>I've retrieved the file if you want to take it down.</p>



<a name="165901643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901643">(May 17 2019 at 13:49)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> cool; is the data workable? :)</p>



<a name="165901693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901693">(May 17 2019 at 13:50)</a>:</h4>
<p>I think so -- working on getting links into the doc now</p>



<a name="165901702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901702">(May 17 2019 at 13:50)</a>:</h4>
<p>Nice</p>



<a name="165901705"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901705" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901705">(May 17 2019 at 13:50)</a>:</h4>
<p>Thanks <span aria-label="heart" class="emoji emoji-2764" role="img" title="heart">:heart:</span> for all your help on this</p>



<a name="165901728"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901728" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901728">(May 17 2019 at 13:50)</a>:</h4>
<blockquote>
<p>Further, I have several questions following. Should we now focus on improving the implementation? Do we know how to improve it? Can we complete such improvement under our current roadmap?</p>
</blockquote>
<p>BTW, <span class="user-mention" data-user-id="220056">@lwshang</span>, I think this is exactly the kinds of questions I hope we will discuss this week! :)</p>



<a name="165901920"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901920" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901920">(May 17 2019 at 13:53)</a>:</h4>
<p>it looks like some crates didn't compile on your machine for whatever reason (e.g., script servo) but we at least have some data</p>



<a name="165901929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901929">(May 17 2019 at 13:53)</a>:</h4>
<p>interestingly it looks like the 14 core machine is... slower? than perf's collection server</p>



<a name="165901946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165901946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165901946">(May 17 2019 at 13:53)</a>:</h4>
<p>i.e. this is the perf vs your machine single-thread (i.e., current master) compiler <a href="https://perf.rust-lang.org/compare.html?start=single-threaded&amp;end=niko-single-thread&amp;stat=wall-time" target="_blank" title="https://perf.rust-lang.org/compare.html?start=single-threaded&amp;end=niko-single-thread&amp;stat=wall-time">https://perf.rust-lang.org/compare.html?start=single-threaded&amp;end=niko-single-thread&amp;stat=wall-time</a></p>



<a name="165902050"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902050" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902050">(May 17 2019 at 13:55)</a>:</h4>
<p>interesting</p>



<a name="165902062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902062">(May 17 2019 at 13:55)</a>:</h4>
<p>Spawning 28 threads probably account for the hello-world overhead. Clock speeds are likely lower too</p>



<a name="165902096"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902096" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902096">(May 17 2019 at 13:56)</a>:</h4>
<p>Then there's some wins due to LLVM using 14 cores</p>



<a name="165902156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902156">(May 17 2019 at 13:56)</a>:</h4>
<p>maybe, yeah</p>



<a name="165902311"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902311" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902311">(May 17 2019 at 13:58)</a>:</h4>
<p>Did niko only bench with 8 threads?</p>



<a name="165902328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902328">(May 17 2019 at 13:58)</a>:</h4>
<p>up to 8 threads</p>



<a name="165902342"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902342" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902342">(May 17 2019 at 13:58)</a>:</h4>
<p>(but even that is quite a bit slower than on perf.rlo, so ...)</p>



<a name="165902394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902394">(May 17 2019 at 13:59)</a>:</h4>
<p>i.e. this is perf 8 threads vs Niko 8 threads: <a href="https://perf.rust-lang.org/compare.html?start=parallel-rustc-8&amp;end=niko-parallel-rustc-8&amp;stat=wall-time" target="_blank" title="https://perf.rust-lang.org/compare.html?start=parallel-rustc-8&amp;end=niko-parallel-rustc-8&amp;stat=wall-time">https://perf.rust-lang.org/compare.html?start=parallel-rustc-8&amp;end=niko-parallel-rustc-8&amp;stat=wall-time</a></p>



<a name="165902487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902487">(May 17 2019 at 14:00)</a>:</h4>
<p>There seems to be a 0.3s startup cost on niko's machine</p>



<a name="165902566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902566">(May 17 2019 at 14:01)</a>:</h4>
<p>I was mostly interested in contention though, probably should use 1 CGU and all cores for that</p>



<a name="165902674"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902674" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902674">(May 17 2019 at 14:03)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span> Did you peek at <a href="https://github.com/rust-lang/rust/pull/60035" target="_blank" title="https://github.com/rust-lang/rust/pull/60035">https://github.com/rust-lang/rust/pull/60035</a> yet?</p>



<a name="165902700"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902700" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902700">(May 17 2019 at 14:03)</a>:</h4>
<p>That would be helpful for incremental+parallel performance, but it's quite large</p>



<a name="165902755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902755">(May 17 2019 at 14:04)</a>:</h4>
<p>only superficially. I figured I'd wait until the half the commits weren't called "wip" anymore :)</p>



<a name="165902777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902777">(May 17 2019 at 14:04)</a>:</h4>
<p>Well I can squash them all together to one =P</p>



<a name="165902790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902790">(May 17 2019 at 14:04)</a>:</h4>
<p>perf results looked promising, but not entirely uncontroversial</p>



<a name="165902831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902831">(May 17 2019 at 14:05)</a>:</h4>
<p>I did review the "preliminaries" PR though</p>



<a name="165902946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165902946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165902946">(May 17 2019 at 14:07)</a>:</h4>
<p>The key idea is to use the same <code>DepNodeIndex</code> across session, this avoids having locks when marking nodes as green, since we no longer need to allocate new indices.</p>



<a name="165903093"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/165903093" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#165903093">(May 17 2019 at 14:09)</a>:</h4>
<p>If you think it's ready to review, I can take a look next week</p>



<a name="166216166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/166216166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lwshang <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#166216166">(May 21 2019 at 21:50)</a>:</h4>
<p>I'm wondering how <code>-j</code> flag of <code>cargo</code> interact with the <code>parallel-rustc</code>.  Correct me if I have any wrong or inaccurate understanding below.</p>
<p>From my intuition,<code>-j</code> controls maximum number of concurrent jobs during <code>cargo build</code>. Such jobs refer to the crates that can be compiled at the same time (no inter-dependency). Meanwhile,  <code>parallel-rustc</code> try to make the compilation inside one crate to be concurrent.</p>
<p>On a N-core machine, we run with <code>cargo build -jN</code>. If at some time point during the build, there are N dependencies can be compiled at the same time (told by dep graph), then roughly all the computation power of the machine is being used. In that case, we may not expect a performance gain by having <code>parallel-rustc</code> enabled and the overhead introduced by the feature may cause a performance regression. When the dep graph makes it impossible to have N concurrent jobs to run, we can expect the <code>parallel-rustc</code> to provide significant performance improvement.</p>



<a name="166246888"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering%20parallel%20data/near/166246888" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/Gathering.20parallel.20data.html#166246888">(May 22 2019 at 08:25)</a>:</h4>
<p><span class="user-mention" data-user-id="220056">@lwshang</span> yes, that is correct.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>